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Abstract 

The National Aeronautics and Space Administration (NASA) Organization Design Team (ODT), consisting of 
twenty seasoned program and project managers and systems engineers from a broad spectrum of the aerospace 
industry, academia, and government, was formed to support the Next Generation Launch Technology (NGLT) 
Program and the Constellation Systems Program. The purpose of the ODT was to investigate organizational factors 
that can lead to success or failure of complex government programs, and to identify tools and methods for the 
design, modeling and analysis of new and more efficient program and project organizations. The ODT conducted a 
series of workshops featuring invited lectures from seasoned program and project managers representing 25 
significant technical programs spanning 50 years of experience. The result was the identification of seven key 
principles of program success that can be used to help design and operate future program organizations. This paper 
presents the success principles and examples of best practices that can significantly improve the design of program, 
project and performing technical line organizations, the assessment of workforce needs and organization 


NAS A/TM— 2008-2 1 4692 


1 



performance, and the execution of programs and projects. The presentations from the workshops from which the 
seven key principles were synthesized are included in the appendices. 


Nomenclature 

ALS 

Advanced Launch System 

CAIB 

Columbia Accident Investigation Board 

CEO 

Chief Executive Officer 

CEV 

Crew Exploration Vehicle 

CLV 

Crew Launch Vehicle 

CONOPS 

concept of operations 

CTS 

Crew Transportation System 

DC-XA 

Delta Clipper Experimental Advanced Vehicle 

DoC 

Department of Commerce 

DoD 

Department of Defense 

EELV 

Evolved Expendable Launch Vehicle 

ESMD 

Exploration Systems Mission Directorate 

FMEA/CIL 

Failure Mode and Effects Analysis/Critical Items List 

FTE 

full time equivalent 

GRC 

NASA Glenn Research Center 

HQ 

NASA Headquarters 

IOC 

initial operations capability 

IPPD 

Integrated Product and Process Development 

IDT 

Integrated Discipline Team 

ISS 

International Space Station 

JSC 

NASA Johnson Space Center 

JSF 

Joint Strike Fighter 

KSC 

NASA Kennedy Space Center 

LaRC 

NASA Langley Research Center 

lh 2 

liquid hydrogen 

lo 2 

liquid oxygen 

LMC 

Lockheed Martin Corporation 

MSFC 

NASA Marshall Space Flight Center 

NASA 

National Aeronautics and Space Administration 

NASP 

National Aerospace Plane 

NGLT 

Next Generation Launch Technology 

NLS 

National Launch System 

NO A A 

National Oceanic and Atmospheric Administration 

NPOESS 

National Polar-Orbiting Operational Environmental Satellite System 

NYU 

New York University 

ODT 

Organization Design Team 

OMB 

Office of Management and Budget 

ORD 

Operational Requirements Document 

SAIC 

Science Applications International Corporation 

SBA 

Simulation Based Acquisition 

SDI 

Strategic Defense Initiative 

SE&I 

systems engineering and integration 

SFOC 

Space Flight Operations Contract 

SpaceX 

Space Exploration Technologies Corporation 

SSC 

NASA Stennis Space Center 

SSTO 

single stage to orbit 

TPS 

thermal protection system 

USAF 

U.S. Air Force 

VSE 

Vision for Space Exploration 

WBS 

work breakdown structure 
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Background 

In January 2003, the NASA established a new program entitled the “Next Generation Launch Technology 
Program” (ref. 1). The purpose of this program was to invest in the development of new technologies that would 
significantly reduce costs and improve reliability and safety for access to space. A Systems Analysis Project was 
created within the NGLT program to systematically evaluate the proposed technologies against conceptual space 
launch architectures for access to space (ref. 2). Architecture analysis teams were established within this project in 
order to analyze the effect the investments made in these technologies, if successful, would have on important 
system-level figures of merit (i.e., safety, cost, reliability, and performance). A series of discipline teams were set up 
to support the architecture teams with a consistent set of ground rules, assumptions, analytical tools and databases. 
In addition to the traditional aerospace technical disciplines (aeronautics, structures, propulsion, etc.) an ODT was 
formed in order to explore the impact that the “human systems” of program, project, and technical line organizations 
have on the space launch “system of systems”. The ODT continued its mission by transitioning to support the 
Constellation Systems Program of the new Exploration Systems Mission Directorate (ESMD), following termination 
of the NGLT program in February 2004. 

This technical memorandum documents the results of the first of three tracks of research undertaken by the ODT, 
namely, research into the lessons learned and best practices established within 25 major aerospace programs of the 
past 50 years. 

Approach 

The ODT established four tracks of inquiry in order to fulfill its charter, as follows: 

Track 1: Survey veteran program/project managers, system engineers, and academics from current and historical 
complex technical programs and projects in order to identify organization related best practices and lessons learned. 

Track 2: Identify tools, methods and databases that could be used by NASA to design, model, simulate, and 
assess future program, project, and line organizations. This included an assessment of current academic research to 
model the human interactions associated with formal organization structure as well as more informal “communities 
of practice” (i.e., networks of technical staff) which typify NASA’s frequent use of small, focused “tiger teams” to 
conduct short term, high impact studies. 

Track 3: Apply the most promising tools and methods identified in track 2 in a series of pilot studies to assess 
their usefulness to future NASA programs and projects, with a particular focus on emerging organizations involved 
in implementing the Vision for Space Exploration (VSE) established by President Bush in February 2004 (ref. 3). 

Track 4: Capture the knowledge developed in tracks 1 to 3 into a “toolkit” to enable a broader dissemination and 
adoption of more rigorous project formulation and organization design best practices across the agency. 

Leadership of the ODT was assigned to NASA Langley Research Center (LaRC), which then recruited a team of 
seasoned program/project managers and system engineers from several NASA field centers, industry, and academia. 
Please refer to table 1-1 in appendix I for a complete list of ODT members who contributed to the workshop process 
described herein. 

The ODT conducted eight workshops during 2003 and 2004, inviting representatives of 25 programs and projects 
spanning the past 50 years to present their thoughts on organizational effectiveness. Each invited speaker provided 
his or her program/project title, objectives, description, and the historical or political context. The presenters were 
also asked to provide a snapshot of the program or project’s life cycle portrayed as a top-level master schedule or 
similar graphical display. Key milestone dates, actual task durations, government (in-house) workforce levels, and 
annual or total budget data were also collected. The briefings summarized the challenges each program encountered 
getting through major decision gates, and, for those programs not restricted by proprietary issues, also presented 
procurement strategies, details of government or customer insight and oversight methods, and the level of 
government integration with contractor teams (such as integrated product teams). A summary of the workshops can 
be found in appendix I table 1-2. 

Key large aerospace programs reviewed included Apollo, the Advanced Launch Systems (ALS) and National Launch 
Systems (NLS), the National Aerospace Plane (NASP), the National Polar Orbiting Environmental Satellite System 
(NPOESS), and the U.S. Air Force (USAF) Evolved Expendable Launch Vehicle (EELV). Smaller and/or 
commercial launch vehicle projects were presented, such as Space Exploration Technologies Corporation (SpaceX) 
Falcon, Delta Clipper Experimental Advanced Vehicle (DC-XA), Kistler and a series of “X” projects. Systems-of- 
systems architectures and systems integration approaches were examined via presentations on the Joint Strike 
Fighter (JSF) Program and the Virginia Class Submarine Program. Other presentations focused on lean 
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organizations, keeping organizations effective over the long term, probabilistic approaches to failure, precursor 
detection and analysis, as well as insights from the Columbia Accident Investigation Board (CAIB) (ref. 4). The 
team also identified research efforts that are underway to improve the ability to assess organizational effectiveness 
and measure organization performance. 

Results and Key Findings 

Using the workshop presentations and team members’ extensive experience, the ODT distilled seven key 
principles that are critical for a program or project to succeed. Though simple in nature, these seven principles 
address recurring issues identified in the workshop presentations and in many past reviews and independent audits 
of significant aerospace and defense programs and projects. The seven principles are summarized in table 1. 

TABLE 1.— SEVEN KEY PRINCIPLES 
OF PROGRAM AND PROJECT SUCCESS 

1 . Establish a clear and compelling vision. 

2. Secure sustained support “from the top”. 

3. Exercise strong leadership and management. 

4. Facilitate wide open communication. 

5. Develop a strong organization. 

6. Manage risk. 

7. Implement effective systems engineering and integration. 


A summary of the programs/projects reviewed by the ODT is presented in table 2, along with an evaluation of 
that program/project against the seven key principles. A cell with a + sign indicates evidence of strong 
implementation of a key principle was found, while a cell with a - sign indicates evidence of weak implementation 
of a given principle was found, and a blank cell indicates that evidence of the principle was not discussed by the 
presenters or uncovered by the ODT. 


+ 

Strong implementation of key principle found 


Weak implementation of key principle found 


No evidence of key principle found 


Each of these principles and the references which support them are explained in more detail in the sections that 
follow. 


TABLE 2.— PRESENCE OF THE KEY PRINCIPLES IN THE PROGRAMS AND PROJECTS REVIEWED 


Program Organizations 

Key Principles of Success (from Table 1) | 

1 

2 

3 

4 

5 

6 

7 

Apollo 

+ 

+ 

+ 

+ 

+ 


+ 

ALS/NLS 


- 

+ 

+ 




DC-XA single stage to orbit (SSTO) 

- 

- 


+ 

+ 

- 

+ 

EELV 



+ 

+ 


+ 


Have Blue/F-1 17A 

+ 

+ 

+ 

+ 

+ 

+ 


JFS 


+ 

+ 

+ 


+ 

+ 

K-l Launch Vehicle 


- 



+ 



NASP 


- 



- 

- 


Nova Super Saturn Ml 



+ 

+ 




NPOESS 

+ 

+ 



+ 


+ 

Space Exploration Initiative (1987 to 1991) 

- 

- 






Space Flight Operations Contract (SFOC) 





- 



Space X Falcon 

+ 

+ 

+ 


+ 



Space Shuttle 


- 


- 

- 

- 


Virginia Class Nuclear Submarine 


+ 

+ 

+ 

+ 


+ 

X-33 

- 

- 


+ 


- 


X-34 








X-37 








X-38 


- 


+ 

+ 

+ 

+ 
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Principle 1: Establish a Clear and Compelling Vision 

Creating a clearly defined vision of the future 
that serves to inspire and motivate the workforce 
is an important first step in paving the road 
toward project success. Top organization leaders 
must clarify the focus of the program goals and 
mission, and clearly enunciate their vision of the 
future to the program team. A vision statement 
should articulate the overreaching, long-term 
goals of the enterprise. It is a look beyond the 
present to see what could be. A vision statement 
should assert what the organization can be at its 
best. It may include a definition of what is unique 
about the enterprise. An effective vision 
statement should be vivid — something one can 
describe — so that people may picture it in their 
minds and then make the connection to how their 
individual contributions can support realization 
of the vision. The statement itself should be 
concise, motivating, and memorable. 

Early in its history, NASA was provided a 
clear vision by President John F. Kennedy 
(fig. 1) to “land a man on the Moon and return 
him safely to Earth.” NASA successfully 
implemented the Mercury, Gemini, and Apollo 
programs to achieve the national vision he 
expressed. Aaron Cohen, former Director of the NASA Johnson Space Center (JSC), noted that during the Apollo 
era, “President Kennedy’s vision was fully complemented with support from Congress and the public, and the 
funding for the project was supported for more than a decade” (ref. 5) 

The unique circumstances at the time made it possible for President Kennedy to receive Congressional and 
popular support for Werner Von Braun’s ambitious plan, to land an American on the Moon before the end of the 
decade of the 1960s (ref. 6) The Apollo program required billions of dollars (~$140 B in 2004 dollars), millions of 
hours, and thousands of men and women, yet, the entire effort was driven by a simple goal: land a man on the moon 
and return him safely to earth by the end of the decade. In the eight years following the first day the idea was 
announced, until Neil Armstrong’s first step on the lunar surface — there remained little doubt among NASA and 
industry personnel about what every meeting, every proposal, every budget discussion, or every decision was 
ultimately intended to accomplish. For almost a decade, President Kennedy’s words served as the guiding spirit, 
pointing the direction for everyone working in the space program. 

The Apollo Program was not the only program example evidencing the importance of establishing a vision. 
Several other successful programs were based on a clear vision and mission. One good example was the F-117A 
and, in particular, its predecessor Have Blue (fig. 5), which was a prototype USAF Stealth Fighter designed, built 
and flight-tested for IVi years during the 1970s. Sherman Mullin, retired president of Lockheed Martin Corporation 
(LMC) Skunk Works, provided an overview of the Have Blue Program, and stated that Have Blue was successful, in 
large part, due to a total buy-in of the vision by both the developer (LMC) and the customer (USAF) (ref. 7) 

The NPOESS Program is another example. This program benefited from a clearly defined vision (ref. 8). Despite 
the fact that NPOESS is a tri-agency effort (Department of Commerce (DoC), Department of Defense (DoD), and 
NASA), with all the political difficulties that this might have entailed, the NPOESS has been successful in 
combining environmental satellite activities from each organization. The vision has allowed the NPOESS member 
agencies to put the broader system needs above their own parochial needs to the benefit of all. 
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Several additional workshop presentations indicated 
that a clear vision and mission are also important to 
less complex programs, and correspondingly, smaller 
organizations. According to Gwynne Shotwell of 
SpaceX, “SpaceX will be demonstrating low cost 
access to earth orbit primarily because the owner, 
financier, and chief executive, Elon Musk, has set a 
clear vision and established a small team of bright 
experienced aerospace engineers (ref. 9). Although 
drawing upon a rich history of prior launch vehicle 
and engine programs, SpaceX is privately developing 
the Falcon I and Falcon V rocket family (fig. 2) from 
the ground up. This private development will include 
the main and upper stage engines, the main engine 
turbo-pump, the cryogenic tank structure and the 
guidance system.” 

On the other hand, a lack of vision can be 
disastrous. While the highly successful lunar missions 
were being performed, President Richard Nixon rejected a vision for a post- Apollo era that involved full 
development of low Earth orbit, permanent outposts on the Moon, and initial journeys to Mars, as far too costly. 
Nixon had no established vision for space exploration, and none was successfully established in the interim until 
promulgation of the current VSE. There had been several subsequent attempts during the 1970s, ’80s, and ’90s to 
establish a clear vision for the space program, but none of these have proven to be successful. For example in 1986, 
the National Commission on Space proposed “a pioneering mission for 21st-century America: To lead the 
exploration and development of the space frontier, advance science, technology, and enterprise, and build 
institutions and systems that would make accessible vast new resources and support human settlements beyond 
Earth orbit, from the highlands of the Moon to the plains of Mars” (ref. 10). Despite some initial enthusiasm for this 
vision, the Commission’s recommendations were never fully embraced by the Reagan administration, which had 
already embraced two major, expensive aerospace projects, the NASP and the Space Station Freedom, and which 
was grappling with replacing the Challenger Space Shuttle Orbiter and returning the Shuttle system to flight. 

In 1989, on the 20th anniversary of the first lunar landing, President George H.W. Bush again proposed a Space 
Exploration Initiative, calling for “a sustained program of manned exploration of the solar system” (ref. 11), but 
once again follow-up support failed to materialize due to competing budget priorities and a lack of buy-in by the 
Congress. NASA efforts on the International Space Station (ISS) were in the early stages of development, and the 
Station was viewed as a higher priority project needed to solve the human health and risk challenges associated with 
human space flight beyond low Earth orbit. The lack of a long-term consensus vision caused near-term goals of the 
shuttle and the ISS to take precedence, and the Initiative languished for lack of funding support. 

Space advocates have been consistent in their call for sending humans beyond low Earth orbit as the appropriate 
objective of U.S. space activities. Review committees as diverse as the 1990 Advisory Committee on the Future of 
the U.S. Space Program (chaired by Norman Augustine), and the 2001 ISS Management and Cost Evaluation Task 
Force have suggested that the primary justification for a space station is to conduct the research required to plan 
missions to Mars or other distant destinations. The Augustine Committee noted, “It seems that most Americans do 
support a viable space program for the nation — but no two individuals seem able to agree upon what that space 
program should be” (ref. 12). 

In several other NASA programs it was not clear if a well defined vision was ever established. These programs 
included the DC-XA, X-33, NASP, ALS/NLS, and the Space Launch Initiative. Although it is true that these 
programs developed a range of exciting technologies — including breakthrough approaches such as hypersonic 
propulsion — these programs essentially failed in their primary goal of bringing into being a next generation space 
launch system to replace the shuttle. A reason for each of these failures would appear to be the lack of a clear vision 
and mission. As pointed out in the DC-XA briefing to the ODT by Dan Dumbacher, NASA DC-XA Program 
Manager, “The DC-XA was a flight test vehicle that was designed to significantly reduce the cost of access to 
space, but, in the long run, NASA inherited this project, the SSTO [single stage to orbit] technology and the SSTO 
vision from the U.S. Air Force” (ref. 13). This refocusing of the program on SSTO performance and reduced 
ownership costs was a major contributor to the X-33 program cancellation in 2001. 



Figure 2. — SpaceX Falcon V Rocket. 
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Ultimately, this lack of a compelling, guiding 
vision of the future has had disastrous 
consequences for the U.S. space program. In 2003, 
the CAIB published its findings relating to both the 
physical and organizational causes of the Space 
Shuttle Columbia accident. The CAIB’s statement 
on organizational contributing factors, excerpted in 
figure 3, summarizes the importance of vision to 
program and organization success. The Board 
noted the lack, over the past three decades, of any 
national mandate providing NASA a compelling 
mission requiring human presence in space. The 
Board attributed NASA’s failure to receive 
budgetary support adequate for its ambitions as consistent with this lack of national vision (ref. 14). The result, as 
noted throughout Part Two of the CAIB Report Volume I, was an organization straining to do “too much with too 
little.” The second reality, flowing from the lack of a clearly defined long-term space mission, was the lack of 
sustained government commitment over the past decade to improved U.S. access to space via development of a 
second-generation space transportation system. Without a compelling reason to do so, successive Administrations 
and Congresses have not been willing to commit the billions of dollars required to develop such a vehicle. 

In order to respond this overarching CAIB finding, on January 14, 2004, President George W. Bush delivered a 
speech at NASA Headquarters (fig. 4) in which he laid out a compelling future vision of human exploration of the 
solar system (ref. 3). After receiving this call to action, NASA immediately reorganized its Headquarters “codes” 
into “mission directorates” to increase agency focus on this future. The centerpiece of this realignment was the 
creation of an ESMD, which was chartered with the task of identifying the requirements and developing the new 
systems that will be required to accomplish the 
missions and implement the vision. 

In the two years since this VSE was 
promulgated, it has served to galvanize and 
refocus NASA and industry on the future. As of 
early 2006, NASA has moved aggressively with a 
three pronged strategy to implement VSE. This 
strategy includes: 

• Return the Space Shuttle safely to flight 
and use it solely to complete construction 
of the ISS. 

• Retire the Shuttle no later than the end of 
calendar 2010 to free up a budget “wedge” 
in excess of $4B annually for use in devel- 
oping future exploration systems, such as a 
new heavy lift Ares V Cargo Launch Vehi- 
cle, an Earth Departure Stage, and a Lunar 
Surface Access Module for returning hu- 
mans to the lunar surface. 

• Initiate immediate development of a new 
Crew Transportation System (CTS). This 
CTS will feature two new systems, the 
Orion Crew Exploration Vehicle (CEV) 
and the Ares I Crew Launch Vehicle 
(CLV). 

Only time will tell how successful the VSE will 
be in pulling NASA, industry and the nation 

toward the future. Figure 4. — President Bush establishing the Vision for Space 

Exploration in January 2004. 
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“This cause of exploration and discovery is not an 
option we choose; it is a desire written in the human 
heart. ” 


President George W. Bush 
February 4, 2003 

“1/1/e leave as we came, and God willing as we shall 
return, with peace and hope for all mankind. ” 

Eugene Cernan (Commander of last Apollo mission) 
December 17, 1972 

“. . . America will make those words 
come true. ” 

President George W. Bush 
January 14, 2004 



Organization Causes of the STS-107 Failure: 

The organization causes of this accident are rooted in the 
Space Shuttle Program ’s history and culture, including the 
original compromises that were required to gain approval 
for the shuttle program, subsequent years of resource con- 
straints, fluctuating priorities, schedule pressures, mis- 
characterization of the Shuttle as operational rather than 
developmental, and lack of an agreed national vision for 
human spaceflight. 


Figure 3. — CAIB’s finding regarding lack of vision (under- 
line emphasis added). 




Summary 


Establishing a clear, concise, and compelling vision of a future state in which the overarching program or project 
goal has been accomplished is a prerequisite to success. The history of the aerospace age is full of examples of the 
power of this principle when successfully implemented, and of the failure that can result when it is not. 

Principle 2: Secure Sustained Support “From the Top” 

The second key principle for program success is to sustain support “from the top.” Sustaining support for large 
programs requires organizations to develop and maintain program sponsors or “protectors”, including protectors 
outside of the organization at the highest levels (e.g., Congress, the White House, Chief Executive Officers (CEOs), 
and the like). For a NASA program, the management team must develop the overall program plan, the program must 
be advocated and “sold” to Congress, the Office of Management and Budget (OMB), and other government 
agencies, and funding must be obtained. The manager must develop effective working relationships with key 
stakeholders at NASA Headquarters (HQ) to keep the program sold, and at NASA field centers to maintain 
enthusiastic internal support, and keep the program staffed with the best available talent. Finally, the manager must 
stabilize the program from distractions and continual replanning, and protect the program from outside interference. 

Protectors are applicable to any size organization or program, and for very large organizations, this could mean 
developing or maintaining contacts as high as the White House. According to Sherman Muhin the internal protector 
function for the Have Blue project was performed by the home division CEO (ref. 7). Special financing was set up 
to protect the secrecy of the special project for which the “Skunk Works” is famous. This top cover allowed the 
project to be autonomous and also prevented other interests at LMC from interfering with the work in progress. 

Programs that successfully cultivated top level 
protectors leveraged that protection to achieve long term 
stability and duration to span multiple congresses and 
presidential administrations. Examples of successful 
long-term programs include Apollo, JSF, Virginia Class 
Submarine, and the F-117A (fig. 5). According to 
Aaron Cohen, Apollo program “protectors” included 
Presidents Kennedy and Johnson, NASA Administrator 
James Webb, and certain high-ranking Senators and 
Congressman (ref. 5). Apollo did not survive President 
Nixon, clearly demonstrating the consequences to a 
major program once top cover protection is lost. The 
F-117A program also spanned more than one 
administration, and its protectors included Senators, 
congressional staffers, senior DoD officials, the USAF, 
and the LMC CEO (ref. 7). According to Paul 
Wiedenhaefer, Paul G. Kaminski, Undersecretary of 
Defense for Acquisition and Technology, served as 
JSF’s “godfather” in the early days of the program, 
enabling the program to bypass many established DoD 
weapons system acquisition practices and thereby 
setting in motion current acquisition reform practices 
(ref. 15). 

The lack of “protectors” has caused many programs 
to fail or to be cancelled. The X-38 project that could 
not survive changes to the project requirements due to 
lack of a top level protector (ref. 16). Also, both the 
ALS and NLS programs suffered from a lack of protectors. These programs had their beginnings in President 
Reagan’s Space Defense Initiative (SDI), which called for a very large booster. However, when President Bill 
Clinton took office, it was determined that the SDI would be de-emphasized and the heavy lift requirement was 
eliminated. This led to the cancellation of the program despite five years of intense effort by the major aerospace 
companies (ref. 17). 



Figure 5. — The Have Blue Prototype (top) and the 
F-117A Stealth Fighter (bottom). 
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Summary 


Securing sponsorship at the top level of an organization, and even higher in the political leadership hierarchy, 
enables long term program stability required for a complex technical aerospace development life cycle to be 
completed and the mission accomplished. 


Principle 3: Exercise Strong Leadership and Management 

Exercising strong leadership is the third principle of success. It requires the leader to: 

• Identify and develop other leaders and technical staff within the organization, 

• Define clear lines of authority and demand accountability, 

• Implement sound project management practices, and 

• Demonstrate uncompromising ethical standards. 

(a) Develop key leaders and technical staff. The first step in building an organization is to assemble a staff with 
the experience and expertise needed to meet the goals and objectives of the program. As Von Braun, a key leader of 
the Apollo Program often emphasized, these individuals should have capabilities in their field of expertise, far 
greater than the program leader(s) (ref. 18). In short, hire people smarter than you and give them the responsibility 
and resources needed to accomplish the task. 

Aaron Cohen provided the following insight into the Apollo management team: “The management structure for 
the Apollo program was successful due in large part to very strong leadership in Washington (George Mueller), 
strong center directors at NASA Marshall Space Flight Center (MSFC) (Werner Von Braun), JSC (Robert Gilruth), 
and NASA Kennedy Space Center (KSC) (Kurt Debus). There were also very strong program managers at HQ (Gen. 
Sam Phillips), MSFC (Everhardt Rees), KSC (Rocco Petrone), and JSC (George Eow). Once the basic decisions 
such as lunar orbital rendezvous (FOR) were made, interfaces between the launch vehicle, spacecraft, and the launch 
complex became very clear and unambiguous. The integration of the Apollo Program was performed out of 
Headquarters with the support of General Electric, Bellcom, and later Boeing. MSFC and its contractors performed 
integration of the various stages of the launch vehicle; and the integration of the command service module/lunar 
module was performed by JSC and its contractors” (ref. 5). 

It is also imperative that the key leaders be selected in the formative stages who will then be asked to perform key 
roles over the life of the program. Dan Dumbacher reported a key lesson learned from the DC-XA program. “Those 
who made decisions in the contract evaluation and negotiation phases must live with their decisions; those who 
negotiate should also manage the project over the life of the project, if at all possible. This will ensure that the 
leader(s) will participate in the formative stages of the ‘Program’ and will basically ‘buy into it’ ” (ref. 13). 

(b) Define clear lines of authority and accountability. The historical experiences have shown that, in successful 
programs, the responsibility and decision-making authority are allocated down to the lowest level possible, 
consistent with the individual’s capabilities, but with the leader maintaining ultimate accountability. The objective is 
to clearly define each team member’s responsibilities and deliverables within the organization, then provide them 
with the opportunity to succeed or fail in executing those responsibilities. This makes the organization more agile in 
making decisions and executing technical tasks and frees the senior managers to focus on the program’s major issues 
and problems. Feadership is all about showing the way and working with people along the way. Effective leaders 
realize that everyone has a unique personality and needs to be dealt with respectfully and on a one-to-one basis. 
Although the importance of “open communication” is addressed as a separate subject later in this report, it is also 
emphasized here because good communication is a key factor in the successful management of any program. It all 
starts with the communication of a clear and credible set of program objectives to everyone on the team. 

An important aspect of effective leadership is the establishment of the responsibilities and authorities of each 
organization on the team. Strong leadership is required on both sides of the government- industry “aisle”. The 
F-117A is a good example of a “hard charging company program manager” combined with a “competent, decisive 
military program manager” (ref. 7). Skunk Works program managers were typically demanding and decisive with 
total responsibility and authority. The program managers also had the ability to control team compensation — which 
may not necessarily have been in accordance with company policies — allowing for better team building and team 
excellence. 
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Walter Dankhoff, the M-l booster engine project 
manager, provided another example of the strong 
leadership that existed in the early years of NASA. The 
M-l (fig. 6) was to be a colossal engine designed for the 
first stage of the then-proposed massive Nova launcher. 

The M-l was much larger than the F-l engines used to 
power the Saturn V booster. “From the beginning, the 
government and industry project managers agreed that 
the M-l program should be addressed as one team, in all 
respects. In this way the rocket engineering technical 
expertise of NASA Lewis Research Center (later 
renamed the NASA Glenn Research Center (GRC)) was 
effectively utilized, along with the Aerojet engineers, in 
addressing the many challenges involved in developing a 
very large liquid hydrogen/liquid oxygen (LH 2 /L0 2 ) 
rocket engine. These two project managers also 
appreciated and utilized the ‘know how’ of the MSFC 
engineers; especially regarding the manufacturing of 
large rocket engine hardware. A good example of this 
‘single team’ approach was in the conduct of the 
program reviews. The project held joint monthly 
program reviews of progress, where the government and 
industry team openly addressed the current problems. 

The project leadership also alternated the location of the 
reviews, one month it was held at Aerojet and the next 
month it was held at Lewis. Senior technical, financial 
and contractual personnel from both organizations 
attended (about 20 total) (ref. 19). 

The result was that this team, in a period from 1962 to 1965, designed, fabricated, and successfully tested, at full- 
scale, all of the rocket engine components (L0 2 and LH 2 turbo pumps; gas generator; and the thrust chamber). Based 
on these results, it was concluded that the development, qualification, and operation of a large LH 2 /L0 2 rocket 
engine was feasible. 

(c) Implement sound project management. An important activity of an effective program leader is the 
establishment of close control of schedules and budgets. To allow for this control on a developmental program 
requires the leader to conduct regularly scheduled program reviews. Over the years NASA has developed a standard 
program/project life cycle which features a series of program review milestones that serve as “go - no go” gates 
through which the program must pass on the path to accomplishing the mission. During these reviews all program 
participants (e.g., government, industry, consultants, and other stakeholders) must be represented. The conduct of 
these reviews provides an excellent opportunity for implementing Principle 4, “Open Communication.” For 
example, a strong leader fosters open, honest communication, including bringing in bad news, without retribution 
against the messenger. 

John Muratore, X-38 project manager, emphasized that strong project leaders should resist the rush to flight until 
all technical and safety issues have been resolved (ref. 16). The X-38 management team delayed their flight test 
program to allow for more aerodynamic analysis to be 
completed. When the aerodynamic analysis was 
completed and there was agreement from a 
independent review team, the project performed the 
next flight test. Ultimately, the project had two perfect 
flights of the vehicle (fig. 7). 

Workshop speakers reported on a number of 
innovative techniques that NASA could adopt to 
improve the rigor of its project management. These 
included use of probabilistic methods in predicting 
schedule performance (ref. 20), use of analytical 
techniques to optimize and “right size” project 
organizations (refs. 21 and 22), and use of project 



Figure 7. — The X-38 drops from the B52 pylon. 
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“simulation” to identify sources of project risk that warrant management attention (ref. 23). These techniques were 
examined in detail by the ODT in tracks 2 and 3 of its research, and are summarized in the post script at the end of 
this memorandum. 

(d) Demonstrate uncompromising ethics. A strong leader must demonstrate uncompromising integrity and 
ethical standards of behavior in order to build respect team member respect and commitment to achieving the goals 
of the program. Team members will not follow a leader who they know is capable of unethical behavior and 
decision making. This behavior on the part of the leader will also foster cynicism within the rank and file of the 
organization, compromising the ability to achieve the mission. 

Summary 

Strong leadership is another prerequisite for success, requiring the program/project manager to identify and 
develop other leaders and technical staff within the team, to define clear lines of authority within the project 
organization and then demand accountability from the designated leads, to implement sound project management 
practices especially with respect to budget and schedule visibility, and to demonstrate uncompromising ethical 
standards in all matters. 


Principle 4: Facilitate Wide Open Communication 

The fostering of open communication has always been a cornerstone of good project management. This 
communication can and has been stifled by leaders who have not been interested in hearing bad news or to be 
bothered with problems. In these instances, the bearer of bad news may avoid trying to communicate problem issues 
to upper management by shifting the problem to others in the organization, even if they do not have the resources to 
handle it. Management must foster open and honest communication without retribution. The organization must be 
open to bad news and be prepared to solve the inevitable problems that always occur. The ODT found evidence of 
good communication in several programs, particularly in the DoD programs: Virginia Class Submarine, JSF, 
F-l 17A, EELV, and ALS/NLS. 

William Starbuck of New York University (NYU) identified several reasons why organizations suppress 
communication and have trouble learning from both success and failure (ref. 24). Organizational learning can 
produce benefits, but many organizations have trouble learning from success. They tend to over-learn successful 
behaviors and become overconfident. Success rigidifies behaviors and limits awareness of environmental changes. 
One result is that future failures become inevitable, and when these serious problems develop and warrant new 
behaviors, organizations have difficulty unlearning 
what has become ingrained. “Success leads to 
specialization and exaggeration, to confidence and 
complacency, to dogma and ritual.” On the other 
side of the fence, organizations also have trouble 
learning from failure — both small and large. They 
have strong tendencies to explain away failures as 
being idiosyncratic and to overlook possible 
systemic causes. This tendency has been referred to 
as the “normalization of deviance” by author Diane 
Vaughn and others in examining the Space Shuttle 
Challenger accident (ref. 25). Indeed, faulty 
communication was found by the CAIB to have 
contributed to the loss of the Space Shuttle 
Columbia (fig. 8). 

A true leader encourages the rapid reporting of 
bad news and is prepared to allocate the appropriate 
resources to solve these issues as they arise. This is 
possible only if all members of the organization are 
sure that they can freely discuss any issue, good or bad, and not be punished for being the messenger. Such an open 
discussion approach can only be fostered by setting an example in meetings and in one-on-one interchanges. All 
members of the organization must know that management is available and receptive to communication at any time. 
Closed doors will often lead to closed communications. By literally having an open door and allowing team 


Organization Causes of the STS-107 Failure: 

The organization causes of this accident are rooted in the 
Space Shuttle Program ’s history and culture . . . Cultural 
traits and organization practices detrimental to safety 
were allowed to develop, including: reliance on past suc- 
cess as a substitute for sound engineering practices (such 
as testing to understand why systems were not performing 
in accordance with requirements); organization barriers 
that prevented effective communication of critical safety 
information and stifled professional differences of opin- 
ion; lack of integrated management across program ele- 
ments; and the evolution of an informal chain of com- 
mand and decision making processes that operated out- 
side the organization ’s rules. 

Figure 8. — CAIB’s finding regarding communication factors 
(underline emphasis added). 
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members to interrupt when communication is required, management will enhance the communications stream that is 
vital for organizational success. There will, of course, be exceptions to this policy at times, but to keep the lines of 
communication open, managers would do well to keep the doors open. 

Walter Hammond went a step further in his presentation, noting that the organization needs to create and 
continuously nurture an “open” corporate culture. Participation enhances understanding, acceptance, and the ability 
to communicate openly and freely. However, Hammond warned against arbitrarily bringing people together to force 
them to communicate. Forced communication could create communication ambiguity, foster the hiding of bad news, 
and unintentionally discourage the team members from discussing grievances (ref. 26). Furthermore, it was noted 
that many strong leaders, by nature of their strength, have a presence that could intimidate those who work for them. 
Leaders should keep this in mind and endeavor to take steps to minimize intimidation so as to avoid the 
development of a “cult of personality” situation that can completely stifle effective communication. 

In today’s workplace, electronic media — mainly in the form of email — has become a dominant form of 
communication. Organizations must face this reality, but must work hard to encourage one-on-one communication. 
This means that the leaders must leave their offices and actively seek out information; it is much easier to get 
information by good, interactive conversation than by an interactive electronic exchange. Face to face meetings 
allow questions and answers to be used to clarify points that would take many emails to accomplish. This is best 
exemplified by David Packard’s “manage by walking around” philosophy: leaving the office and literally walking 
around and meeting with team members to discuss the activities in which they are involved. 

One of the challenges of managing complex technical programs and projects of national scope is geographical 
team distribution. Overcoming the barrier of team members operating remotely from each other requires special 
attention be paid to ensuring open lines of communication. The DC-XA program used daily telecons with all team 
contractors and leaders from the appropriate technical disciplines to identify issues and allow the team to 
immediately formulate a plan to resolve them. Near-term schedules were used as a point of reference in all 
discussions (ref. 13). 

Probably the most important principle that an organization can use to ensure open and free flowing 
communication is to abide by the adage of “praise in public and criticize in private.” Public criticism is the surest 
method of stifling communications. Few individuals will dare to come forward with critical information if it is 
known that this might bring public criticism. On the other hand, it is not suggested that managers forego criticism 
that is required, but rather that they deal with it behind closed doors. 

Summary 

Open communication is the cornerstone of programmatic success. It enables timely identification of issues that 
require the attention of leadership, it facilitates relationship building across the project team, and it sets the tone for a 
team’s culture and identity. Communication barriers can easily take hold in organizations in the absence of 
management attention, and once in place can contribute to organizational failures, with disastrous consequences. 


Principle 5: Develop a Strong Organization 

The fifth key principle of success is development of a strong organization to execute the project. Starbuck 
presented his work on why organizations fail and how organizations change over time. He emphasized that 
organizations can remain effective over long periods if three interdependent pillars — culture, rewards, and 
structure — are carefully designed and aligned (ref. 24). This is a key finding of the ODT workshop process because 
it emphasizes that program managers must expend resources designing and monitoring not only the organization’s 
structure but also its culture and rewards system. These three pillars are discussed in more detail below. 

(a) Culture. Fundamentally, culture consists of the shared beliefs and behaviors of the individuals in the 
organization. Elements of culture include, for example, language, ethics, folklore, dress and behavioral norms. 
Starbuck emphasized that organizational culture typically takes a long time to create, but can be fragile, thus 
requiring periodic and thoughtful management attention. Ideally, an organization’s culture should foster conflict 
management and openness to communication, especially for organizations involved in high risk, technologically 
complex endeavors like NASA. Culture can be influenced using multiple strategies: careful employee selection, on- 
going training, employee performance monitoring, and rewards. According to Starbuck, careful employee selection 
is the most influential strategy among these options (ref. 24). 

The ODT identified the NPOESS as an outstanding example of careful organization design and culture 
management (ref. 27). The NPOESS program intentionally sought to create a new culture unique to their joint 
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program office by colocating personnel from the three contributing agencies, DoD, National Oceanic and 
Atmospheric Administration (NOAA), and NASA in a separate facility removed from their agency home offices. 
They carefully negotiated respective roles and responsibilities and documented them in a tri-agency memorandum of 
agreement before staffing and initiating integrated program office operations. This included identifying in advance 
specific positions in the organization that would be filled by each agency. In this way, they carefully removed as 
many potential organizational conflicts and barriers to success as possible before moving into program execution. In 
addition, management focused the attention of the integrated multi-agency staff members on the joint organization’s 
unique vision and goals, in an effort to foster the emergence of a new organizational culture. 

(b) Rewards. Rewards are used to align the behavior of team members with the organization’s goals and must 
reinforce the organization’s priorities. For example, to get quality, rewards must depend on quality. To get 
communication, rewards must depend on communication. Sherman Mullin cited positive organizational culture as 
one key to the success LMC Skunk Works development of the F-117 (ref. 7). Management endeavored to foster 
desired behavior by action, not just by pronouncement. They particularly used a reward system, sometimes one that 
was outside the normal corporate awards system, to establish a precedent for excellence. In addition, they attempted 
to allow for honest mistakes and failings among the team members, thus fostering investigation and improvisation 
on the part of all the team members and allowing for the creation of a learning organization. 

Starbuck emphasized that rewards (and punishments, if also used) are more effective to change behavior the 
closer they are granted to the observed behavior. NASA, as a government agency, must work within federal 
regulations to design and implement reward programs. Government civil service regulations limit the availability of 
financial rewards and reward timing. Thus, program managers may need to find creative alternative ways to reward 
desired behavior and facilitate culture change. 

(c) Structure. Organization structure consists of two elements: formal line organization which describes direct 
reporting management and supervision pathways, and informal communication and integration structures that 
constitute how individuals in the organization actually work. The latter informal structure in a typical large NASA 
or DoD program consists of a hierarchy of decision making boards informed by technical working groups and 
integration panels. At the lowest level, the organization structure may consist of a set of integrated product, process 
or task teams that are established to execute the day to day activities of the project. The challenge in using a 
“shadow” boards/panels/teams organizational approach is to keep them lean and limited in number. An excessive 
number of boards, panels and teams can confuse the rank and file staff as to who is responsible for executing, 
reviewing and approving the technical work products: the program/project/line organizations, or the shadow 
organization of boards, panels and teams. Boards should rightly be chartered to implement configuration control 
rather than produce technical work products. 

A fundamental principle of organization structure design is to align the organization with the system or product 
architecture, so that personnel understand clearly what they are responsible for delivering. This principle was well 
understood by the leaders of the Apollo program. George E. Mueller, who served as associate administrator for 
manned space flight at NASA Headquarters during Apollo, emphasized the importance of utilizing a common 
organization structure at each tier in the program organization (ref. 28). An example of this principle is shown in 
figure 9, which illustrates how a common organization structure establishes well defined interfaces, and therefore 
communication pathways, between the managers of the same functions at each tier in the program organization. 

The Virginia Class Submarine Program, presented by George Drakeley, deputy program manager, is another case 
example of the value of a strong organizational architecture that is well aligned with the product being produced. 
This program’s structure and approach is an excellent example of the Integrated Product and Process Development 
(IPPD) acquisition methodology developed by the DoD in the 1990s to streamline major weapons systems 
acquisition. In this case, the submarine design and construction teams were organized along three lines (see fig. 10). 
Major Area Teams (MATs) were established to design and produce each major segment of the submarine (e.g., 
pressure hull, engine room, etc.). System Integration Teams (SITs) were established to design and produce those 
systems which are distributed throughout the submarine (e.g., electrical power, hydraulics, etc.). Process Integration 
Teams (PITs) were set up to design and implement common processes such as cost, schedule, risk, and 
configuration management. A single Major Area Integration Team (MAIT) was established to integrate across the 
MATs, SITs and PITs, essentially performing the systems engineering and integration (SE&I) functions via the 
coordination and integration of the various team activities. Three implementation techniques were critical to making 
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Figure 9. — Apollo organization structure with common units at each program tier (ref. 30). 
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Figure 10. — NAVSEA Virginia Class Submarine organizational approach featuring multi-tiered integrated 
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this IPPD structure work: (1) early selection of the design/build contractor team; (2) “badge-less” team membership 
consisting of active duty military, government civilian, and contractor staff; and (3) cross-team membership with the 
MAIT drawing its membership from the MAT, SIT and PIT team leaders as appropriate. The payoffs of this IPPD 
organizational architecture and acquisition approach are well documented in a DoD case study published in 2000 
(ref. 30). These payoffs included: a shortened overall design cycle/schedule, greatly reduced number of change 
orders from less problems encountered during construction, considerable cost avoidance in producing the first 
vehicle, and fielding an operational weapon system that effectively balanced capability and flexibility with cost. 

Starbuck proposed that the most important structural property is communication. Starbuck emphasized that 
people talk upward and listen upward, and they avoid telling bosses what the bosses do not want to hear. Thus, 
managers need to be very sensitive to the signals they put out. In addition, Starbuck talked about the inevitable and 
often frequent conflict between engineers and managers. Specifically, he suggested that organization structures 
should focus on communication channels rather than rules, so that disagreements between managers and engineers 
are useful, rather than divisive (ref. 24). Well designed organizational architectures clearly establish these natural 
communication channels through structuring. The Apollo structure shown in figure 9 accomplishes this by 
establishing standard common functions at multiple organization tiers. The NAVSEA Virginia Class Submarine 
Program organization shown in figure 1 0 accomplishes this by chartering teams composed of members who need to 
communicate across their organizational interfaces in order to accomplish effective system integration. 

Another consistent characteristic of successful project organizations was the use of small teams of individuals, at 
least at the initiation of the programs. Coordination was facilitated, as in the case of the Skunk Works, SpaceX, and 
X-38, by the entire team being located on the factory floor - or at least in the same facility. Elon Musk specifically 
emphasized the development of a small team of high-priced and valued individuals, as opposed to the alternative 
approach of creating a large team of varied-priced individuals that is more characteristic of the broader NASA 
organization and aerospace contractor base. He has attempted to create a cohesive team where individuals are more 
broadly involved in the development process, with cross training and multiple functional responsibilities. 

The authors believe that NASA works at its very best when it uses small “tiger teams” to tackle high priority, 
challenging problems. Most new program formulation in the agency occurs first in a 30, 60, or 90 day tiger team 
study. A prime recent example of this technique was the 60 day Exploration Systems Architecture Study (ESAS) 
(ref. 31). This study was established by NASA Administrator Michael Griffin in May 2005 in order to move the 
VSE into its next phase of development. A primary product of the ESAS study was the architecture of the CTS 
intended to replace the Space Shuttle. The agency’s ability to quickly formulate, staff, and execute high priority 
studies via tiger teams is enabled by the breadth and depth of NASA’s staff, their orientation along technical 
discipline lines, and the informal communication networks that have been built at the working staff level. These 
discipline-oriented “communities of practice” enable communication between experts within each discipline in spite 
of the geographic and cultural barriers which separate the ten agency field centers. 

Summary 

Development of a strong team organization enables program/project success. Three mutually dependent pillars 
support an effective organization: culture, rewards, and structure. Each requires proactive leadership and dedication 
of programmatic resources to carefully design and foster. Communication is an essential ingredient to success in 
each of the pillars, and can be facilitated via careful design of the team structure. Alignment of the organization 
structure with the product or system being produced is important. Use of small “tiger teams” has proven successful 
in formulating new programs and executing fast track projects. 


Principle 6: Manage Risk 

Managing risk involves employing a continuous and evolving risk-management process that is complemented by 
an extensive rapid prototyping program using modeling and simulation, testing, and hardware/software-in-the-loop 
tests. 

NASA has always considered the risk of loss of the crew and loss of the mission in its decision-making. In fact, in 
the early days of Apollo, quantitative goals were established and estimates of both of these risks were required. 
However, during the later Apollo era and the Shuttle era, NASA relied primarily on qualitative measures to estimate 
and control mission success and safety risk (ref. 32). Further, NASA addressed technical and developmental risk 
qualitatively, relying on concepts that had reached a given level of maturity, as measured on a qualitative technology 
readiness level (TRL) scale, to be included in proposed designs. In a similar fashion, cost and schedule risk (the risk 
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of cost overruns or schedule slippages) were addressed by employing bottoms-up, judgmentally-based approaches 
not tied directly to the key technical risk drivers that were often the causes of these undesired consequences. 

Risk assessment in the Shuttle program prior to the Challenger accident was primarily qualitative and 
judgmentally based. Analysis was performed using the bottoms-up approach previously described. This can be seen 
in the Failure Modes and Effects Analysis and Critical Items List (FMEA/CIL) undertaken to address safety and 
mission risk. These analyses proceed from the local, component-related failure issues, to the more global systems 
and vehicle-related issues using deductive logic rather than the inductive logic more characteristic of top-down 
analytical approaches. Cost estimation provides another example. When cost estimation was undertaken, it was 
based either on a bottoms-up actuarial approach or upon mass-driven cost estimating relationships. 

The ODT invited Elisabeth Pate-Cornell of Stanford University to present a lecture on her work. She explained 
that her 1990 study was directed at the problem of the loss of, and damage to, tiles on the belly of the Shuttle Orbiter 
(ref. 33). Since the data used was based upon post- flight observations of the first 33 flights of the shuttle, it included 
stack debris, micrometeoroid and Orbiter debris, as well as landing debris, but it did not address debris strikes to the 
reinforced carbon-carbon used on the leading edges of the shuttle, which was the proximate physical cause of the 
loss of the Columbia orbiter during re-entry. Pate-Cornell’ s study had used this historical data in combination with 
design information concerning the placement of critical systems, and the reentry thermal physics information to 
develop a risk map (see fig. 1 1) for the shuttle belly tiles and to recommend a continuing inspection program related 
to these tiles based upon the combined risk “map” that was produced and on the priorities that it displayed. 

Subsequent to the Challenger accident, and as a result of the Rogers’ Commission Report (ref. 34), and 
particularly due to the efforts of one commission member, the late Richard Feynman, NASA was urged to take a 
more quantitative approach toward risk. In response to these recommendations, NASA established three pilot studies 
to employ quantitative risk assessment using the scenario-based probabilistic risk assessment approach pioneered in 
the 1970s by Norman Rasmussen in the commercial nuclear industry. These three studies addressed the safety and 
mission success risk of the main propulsion pressurization subsystem (performed by Lockheed Palo Alto), the 
auxiliary power unit (APU) (performed by a McDonnell Douglas-Pickhard Lowe and Garrick team), and the thermal 
protection system (TPS) (performed by a Stanford-Camegie Mellon team). This latter study was led by Pate-Cornell. 
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Figure 11. — Shuttle Tile Damage Risk Map (ref. 33). 
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Pate-Cornell indicated that one of the most important lessons learned from her work on the Space Shuttle and 
elsewhere was the importance of collecting operational data on an ongoing basis, and embedding it into a risk-based 
structure to provide an ongoing or “living” measure of the residual risk of continued operation. She cited the crash 
of the Concorde on July 25, 2000, as another example of how this data might have been used to prevent failure. Her 
records research indicated that during the 75,000 hr of previous Concorde operation, 57 tires had burst, and several 
times, debris had come close to penetrating the fuel tank before the crash in 2000. Pate-Cornell pointed out that the 
use of such a “living accident precursor” system had been shown to be of value in such diverse areas as hospital 
anesthesia, the Ford/Firestone Explorer tire failure, and the leading edge of the Boeing 737. 

Pate-Cornell’ s presentation, combined with the ODT’s own study and presentations of the information contained 
in the report of the CAIB, indicated the value of managing safety and mission success risk on a continuing basis 
using quantitative models of potential accident scenarios, combined with developed operational data, and physical 
models of relevant phenomena (ref. 33). 

The JSF Program presentation indicated the value 
of the continual use of modeling and simulation tools 
to track and “buy down” the risk of development 
early on and with rapid prototyping of design 
concepts later (fig. 12) (ref. 15). Combining these 
phenomenological simulation tools and prototyping 
with the logical simulation tools provided by 
quantitative risk assessment may provide a better 
way to assess, monitor, and track developmental risk 
and marshal the scarce test resources available in the 
NASA constrained budgetary and low production run 
environment. 

The X-38 project (ref. 16) recommended picking 
the “top 10” risks and focusing on those with the 

greatest payoff. The program then used these Figure 12,-Lockheed Martin X-35 JSF concept in flight, 
identified risks to guide the “build a little, test a little 

and grow a little” philosophy employed throughout vehicle development to effectively manage the project’s risk. 
The primary focus of the X-38 philosophy was “you can’t do too much modeling and simulation.” However, the 
program recognized that it needed actual data to anchor these model results. Guidelines driving the X-38 modeling 
and simulation were: 1) don’t extrapolate beyond known physics and test assumptions; 2) add complexity only as 
understanding increases, and 3) expect and plan to make modeling and simulation an iterative process. The X-38 
project also believed in the power of frequent integration tests (these can be defining moments for a project) to 
reduce risk. Integration tests push for opportunities to bring things together for early integration and validation and 
to assess how integration is going, without waiting to the end of the project. 

Risk management is critical in the success of all programs — not just manned missions. This insight was 
reinforced by the example of the EELV program presented by Major Andrew Chang from the USAF EELV Program 
Office (ref. 35). After a review of the performance history of their EL Vs in the early 1990s, the USAF determined 
that their experience with launch failures was worse than that of NASA. As a result, in advance of beginning 
development of new EELVs (Atlas V and Delta IV), decisions were made to: 

• Concentrate their space resources under a unified command; 

• Establish an independent readiness review team similar to NASA’s flight readiness reviews; and 

• Establish an independent risk management council and a single government mission director for each indi- 
vidual mission. 

As of 2005, eight flights — five on Atlas V and three on Delta IV — have all been successful. However, this 
achievement was accomplished with the application of considerably increased government resources. The program 
originally started with 150 full time equivalent (FTE) staff, and by FY04 workforce had increased to 350 to 400. 
This lesson learned stands in stark contrast to the “lean” staffing approach and systems development philosophy 
which Adm. Craig Steidle (ret.) brought to the startup of ESMD in 2004. 

Summary 

Rigorous and continuous risk management is critical to the avoidance of system failures that can bring down a 
program. Program leaders must find the appropriate mix of qualitative and quantitative risk assessment techniques, 
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both of which have a strong foundation of technical principles and successful applications at this point in the 
evolution of aerospace systems. Continuously ranking and tracking project risks provides an excellent focus for 
management attention and resources. 


Principle 7: Implement Effective Systems Engineering and Integration 

The final key principle, Implement Effective Systems Engineering and Integration, is composed of several 
subprinciples: 

(a) Develop clear, stable objectives and requirements from the outset; 

(b) Establish clear and clean system interfaces; 

(c) Maintain effective configuration control; 

(d) Use modem information technology and analytical tools to model and simulate system performance, includ- 
ing organizational performance, well in advance of hardware development. 

Aerospace programs are typically developmental in nature. Sometimes they are revolutionary in scope. The very 
nature of developmental programs implies that the outcome is, at least to some degree, uncertain. In such programs, 
shifts in objectives result in a corresponding growth in requirements. These shifts and the resulting requirements 
growth lead to program delays, cost increases, and even to program failure or cancellation. In successful programs, 
the traditional role of SE&I has been to stabilize the development process and to keep the program under control. A 
successful SE&I effort is characterized by the initial establishment of a clear and stable set of objectives for the 
program at the outset, and the careful and thoughtful development of a minimal set of requirements for the 
achievement of those objectives. 

(a) Develop clear, stable objectives and requirements from the outset. Good systems engineering starts with 
good requirements and good management practices that will result in clear, stable program objectives. The managers 
of all the programs presented at Workshop VI (see appendix F) stressed the need to develop clear, stable 
requirements from the outset and to take the necessary time on the front end of the program to develop, stabilize, 
and build consensus. Several programs (F-l 17A JSF, SpaceX, X-38) proposed that top-level program requirements 
be limited to one page, and all agreed that the focus should be on achieving key driving requirements. The F-117A 
program (ref. 7) had five requirements: (1) To provide program security, (2) To meet low observable specification 
(Stealth); (3) To have precision guided weapons capability; (4) To deliver a 5000 lb payload; and (5) To achieve 
initial operations capability (IOC) by the specified target date. 

The JSF program developed and employed a strategy-to-task-to-technology approach to prioritize needed 
technologies. Based on the Operational Requirements Document (ORD) and concept of operations (CONOPS) 
model, the program defined operational capabilities and had designers, operators, and technologists work together to 
identify and prioritize mission-enabling technologies. The program invested billions into the resulting technologies. 
Not all technologies required a huge investment, but some did require a change in philosophy. The example, 
discussed by Paul Wiedenhaefer, JSF Requirements Lead, was Avionics Open Architecture and supporting 
technologies to address affordability (ref. 15). 

John Muratore, the X-38 program manager (ref. 16) recommended against writing documents or specs just to fill 
in the squares (volumes get shelved that no one will ever read or use) and recommended careful control and tracking 
of interfaces. “Your strengths can also be your weaknesses.” He summarized that requirements traceability can be a 
good value, but it can also be a program’s downfall. Traceability itself is not the goal; the key is to employ a 
traceability tool that will improve the product or the process. 

Successful programs, such as Apollo and JSF demonstrate that it takes considerable initial effort and time to 
develop the set of key driving requirements, but that this significant effort more than pays for itself during the course 
of later program development. Paul Wiedenhaefer related that the JSF ORD took over five years to develop while 
rapid system prototyping (i.e., fly-offs) was occurring. The Program Office also developed, maintained, and matured 
a CONOPS Model to evaluate contractor developed architectures and spent significant effort in achieving consensus 
around the cost models to reduce future challenges of cost analysis and modeling results. 

(b) Establish clear and clean system interfaces. SE&I performs a critical role early in the design effort in the 
establishment of interfaces between the discrete elements in a properly defined system and the hardware and 
software being developed to implement the design requirements. The SE&I team must clearly delineate boundaries 
between subsystems, document them in a concise interface control document, and continually monitor design team 
compliance with these documents throughout the design process. 
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In addition to the need for clean product/system interfaces, many presentations throughout the workshops 
emphasized the importance of establishing clear and clean organization interfaces. This includes documenting 
responsibilities, authorities and accountability of the organizational units on each side of the interface so as to drive 
out a common understanding and operating agreement. Specifically, they emphasized the benefits derived from 
designing the program organization structure to mirror the system architecture, with particular emphasis on the areas 
of the design that were nontraditional or developmental. Skunk Works Program Manager Sherman Mullin reiterated 
this point several times in his presentation (ref. 7). Program experience relative to other NASA and other 
government programs identified the importance of allowing the contractor the freedom of structuring its 
organization to keep the interfaces clean and the accountability clear. In particular, government-enforced contractor 
teaming was to be avoided because it tended to obscure the lines of authority and to complicate the structure of the 
program organization. An interesting contrast to this view was seen in the Virginia Class Submarine Program, which 
implemented a congressionally directed teaming arrangement between Northrop Grumman Newport News 
Shipbuilding and Electric Boat as the joint design/build prime contractor for this next generation submarine. The 
Virginia Class program mitigated the potential organizational and system interface complexity that could have 
resulted from this teaming via the IPPD structure described under Principle 5 above. Virginia Class deputy program 
manager George Drakely emphasized three implementation techniques that aided in mitigating this teaming risk: (1) 
forming “badge-less” integrated govemment/contractor MAT/SIT/PIT membership, (2) establishing crisp program 
content responsibilities for each contractor, and (3) forging an agreement to split all profit evenly between the two 
contractors. 

(c) Maintain effective configuration control. Once top level objectives have been agreed to and sold to the 
program’s protectors and customers, once these objectives have been decomposed into a set of system requirements, 
and once the program organization has established clear, clean interfaces both for itself and for the conceptual 
system design, then this programmatic “baseline” configuration must be carefully controlled. Several presenters 
emphasized the need for establishing an early program baseline, consisting of requirements, system concept, 
budgets, schedules, and risks, then rigorously controlling that baseline as the program moves forward through its 
lifecycle. The SE&I effort must keep the design team on track, focusing on the key design requirements and being 
continually vigilant against “requirements creep”. Shifting requirements ultimately led to the demise of the National 
Aerospace Plane program, which started in 1984 as a single stage to orbit technology demonstrator but which ended 
as a collection of various hypersonic technology development activities at its termination in 1993 (ref. 36). An 
uncontrolled budgetary estimate contributed to the failure of the Space Exploration Initiative under the first Bush 
presidential administration. 

(d) Use modern information technology and analytical tools to model and simulate system performance 
well in advance of hardware development. In the past, the SE&I effort was very labor intensive. System engineers 
were required to develop a peer level understanding of each of the developing designs within each area of the work 
breakdown structure (WBS) and review these designs against each other. The design reviews were lengthy, 
comprehensive, effective, but expensive. Recent advances in software (e.g., engineering analysis tools), hardware 
(e.g., computing platforms), and information processing (e.g., the Internet) have revolutionized the engineering 
design process. Presently, it is possible to construct computational simulations of system designs to enable many 
alternatives to be examined during trade studies. Additionally, the use of web-based information management tools 
can now allow the geographically distributed members of the design team to have access not only to the drawings 
and specifications of the key design alternatives as they develop, but also to the results of supporting test and 
simulation data. 

There were many programs presented at Workshop VI (JSF, X-38, SpaceX, see appendix F) that identified 
simulation based acquisition (SB A) as critical to a program’s success. Two outstanding examples of SB A surveyed 
by the ODT came from the Navy: the JSF program and the Virginia-Class Submarine program. The JSF program 
invested upwards of $1B implementing SB A to allow thorough requirements-cost tradeoffs, technology assessment, 
and virtual system design prior to initiating hardware production (ref. 15). The Virginia-Class Submarine program 
utilized a single electronic database to integrate all aspects of design, planning, and construction. The database was 
used to link design with production and business operations, providing a fully integrated electronic data set 
throughout the lifecycle of the program (ref. 29). In addition, the effectiveness of early developmental 
hardware/software-in-the-loop testing was enhanced by the existence of this common electronic database. 

Summary 

Effective system engineering and integration has been repeatedly identified in case histories and mishap 
investigations as a critical enabling discipline for program and project success. Effective SE&I can be brought about 
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by developing clear, stable project objectives and requirements during formulation phase, establishing clear and 
clean system and corresponding organization interfaces, maintaining effective control of technical, cost, and 
schedule configuration baselines, and using modem information technology and analytical tools to model and 
simulate system performance, including organizational performance, well in advance of hardware development. 


Postscript — Tracks 2, 3, and 4 

This report documents the ODT’s first track of inquiry. Three additional tracks of inquiry were subsequently 
undertaken by the ODT. Track 2 focused on identifying tools, methods and databases that could be used by NASA 
to design, model, simulate, and assess future program, project, and line organizations. In track 3, the ODT applied 
the most promising tools and methods identified in track 2 to a series of pilot studies to assess their usefulness to 
future NASA programs and projects, with a particular focus on emerging organizations involved in implementing 
the VSE. Track 4 has focused on development of a “toolkit” to capture the results of the first three tracks in a web- 
served software application for adoption by a broader set of users in the agency. The key findings of tracks 2 to 4 are 
synopsized below. 

The ODT collaborated with Starbuck and his staff at NYU to participate in a National Science Foundation 
sponsored Workshop on Organization Design (ref. 37). Presentations at this workshop provided insight into the 
current state-of-the-art of Program/Organization Modeling and Simulation (POMS). The premise of POMS is as 
follows: Program managers need to organize human discipline experts into high performance groups, typically 
separated by geographic, line organization, program organization, or technical discipline boundaries to meet the 
unique technical, schedule, budgetary, and other requirements of a program. Thus, the ability to more effectively 
design, model, and analyze the human interactions typical of complex program organizations might well prove to be 
an enabling discipline to future successful program execution (ref. 38). 

As a result of the NYU workshop, presentations at the ODT workshops, and research performed under ODT 
sponsorship, five key POMS tools have been identified and demonstrated in pilot studies by the ODT to date. These 
are: 

1. Historical Program/Project Database. The ODT compiled a database of over 40 historical programs and 
projects for the purpose of establishing Manpower Estimating Relationships which correlate peak year 
workforce levels to total program funding, as well as historical program life cycle phase durations which can 
be used to feed stochastic assessments of schedule risk for future programs and projects (ref. 20). 

2. OrgCon (EcoMerc, http://www.ecomerc.com/). OrgCon is an expert system desktop software tool which 
incorporates 350 rules derived from the multi-contingency organization theory. OrgCon is used to examine the 
alignment or “fitness” of an organization design’s internal structure to its culture, constraints and external 
environment. It can also be used to perform sensitivity studies to optimize the organization design by 
minimizing “misfits” amongst the 14 fitness factors that OrgCon uses to characterize an organization. The 
ODT has applied OrgCon to the design of the Constellation Systems Analysis Integrated Discipline Team 
(IDT) (ref. 23). 

3. Design Structure Matrix. The Design Structure Matrix (DSM) was developed by the Massachusetts Institute of 
Technology to solve the problem of graphical representation of project task flows where tasks may be iterative 
or require feedback. For application to organization design, it can be used to quantify the interdependence 
and/or information/product flows between organizational units. Tim Brady of JSC used the DSM technique to 
analyze the risks that contributed to the failure of the Mars Climate Orbiter and Mars Polar Lander in 1999 
(ref. 21). The ODT subsequently applied the DSM to successfully design a set of fourteen IDTs used by 
ESMD Constellation Systems to organize over 200 FTE across all ten NASA field centers for executing the 
FY05 program (ref. 22). 

4. SimVision (e Project Management LLC, http://www.epm.cc/index.htm). SimVision is a discrete event 
simulation tool based on information-processing theories. It uses an “agent” based modeling architecture to 
simulate workers (agents) performing tasks. SimVision was originally developed at Stanford University and 
has over 15 years of verification and validation heritage. The tool can then be used to right-size organizations, 
balance workforce assignments, and estimate multiple measures of organization and schedule risk for use in 
optimizing the final organization design. The ODT has used SimVision in multiple studies to design and 
simulate project team organizations, products, tasks, and schedules (refs. 23 and 40). 

5. NASA Project Manager’s Toolkit. In FY05, the ODT initiated development of the Project Manager’s Toolkit, 
a web-served software application intended to capture the knowledge developed by the ODT since FY03, and 
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provide the working program/project manager and system engineer/analyst with desktop access to these tools 
and techniques (ref. 39). 

In summary, the ODT has identified and demonstrated several organization design and analysis techniques that, if 
used in combination with the seven key principles of program success, can significantly improve the agency’s ability 
to field more effective program, project and line organizations for the future. 
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NASA Langley Research Center, March 2004 
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Workshop VII: Writing the Report 
NASA Langley Research Center, August 2004 
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Appendix I 

ODT Membership and Workshops 

TABLE I- L— MEMBERS OF THE ORGANIZATION DESIGN TEAM 


Name 

Organization 

Title 

Vincent J. Bilardo, Jr. 

NASA Glenn Research Center 

Co-lead of Organization Design Team, GRC 
Crew Launch Vehicle Project Office 

John J. Korte 

NASA Langley Research Center 

Co-lead of Organization Design Team 

Dallas Bienhoff 

The Boeing Company 

Chief Architect, Space Exploration Systems 

Darrell R. Branscome 

SAIC Hampton, Virginia 

Senior Systems Engineer 

David Cheuvront 

NASA Johnson Space Center 

Senior Systems Engineer, Exploration 
Systems Engineering Office 

Robert L. Chick 

The Boeing Company 

Lead, Boeing Best Practices 

Walter Dankhoff 

SAIC/Space Propulsion Synergy Team 

Chairman, also Retired NASA HQ 

Freddie Douglas, III 

NASA Stennis Space Flight Center 

Head, Systems Management Office 

Dale Dugal 

Hernandez Engineering Incorporated 

Senior Product Assurance Engineer, NASA 
MSFC 

Joseph R. Fragola 

SAIC New York 

Principle Scientist 

Thomas J. Gormley 

Gormley & Associates 

President 

Walter E. Hammond 

Jacobs Engineering, Sverdrup 

Senior Systems Engineer, Support NASA 
MSFC 

James J. Hollopeter 

Earth Space Applications 

President 

Kevin J. Langan 

SAIC 

Senior Systems Engineer 

Albert E. Motley, III 

NASA Langley Research Center 

Senior Systems Engineer, Systems 
Management Office 

Deborah J. Neubek 

NASA Johnson Space Center 

Deputy Manager, Advanced Design Office 

Mark Prill 

NASA Marshall Space Flight Center 

Head, Technology Requirements Planning 

Randy Sweet 

Lockheed Martin Corporation 

NGLT Deputy Program Manager 

Alan Wilhite 

National Institute of Aerospace 

Langley Distinguished Professor 

Keith L. Woodman 

NASA Langley Research Center 

Systems Engineer, Space Access & 
Exploration Program Office 
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TABLE 1-2.— ORGANIZATION DESIGN TEAM WORKSHOP SUMMARY 


Workshop 

Location, date 

Title 

Highlights 

I 

LaRC, 
August 2003 

Tools and Methods for Organization 
Design & Analysis 

Five invited talks. Dependency Stmcture 
Matrix (DSM) tool for evaluating 
organizations. 

II 

JSC, 

October 2003 

Advanced Mission Operations and 
Analysis 

Five invited talks focused on autonomous 
space flight as means of reducing 
organization costs. 

III 

Williamsburg, VA. 
Decemenber 2003 

Organizational Design and Best 
Practices 

Eight invited talks focused on 
program/organization design. 

IV 

SSC, 

February 2004 

Layout of the Next Program 
Organization Structure 

Prototyping an organization design 
process. 

V 

LaRC, 
March 2004 

Layout of the JSF-Like Program 
Structure 

Project life-cycle definition and FTE 
estimation process. 

VI 

JSC, 

May 2004 

Building a Historical Program 
Database 

Twelve invited talks focusing on program 
best practices and lessons learned. 

VII 

New York University, 
June 2004 

Conference on Organization Design, 
co-sponsored by NYU Stem School 
and National Science Foundation 

Key org design tools (OrgCon, SimVision) 
identified for application in pilot studies. 

VIII 

LaRC, 
August 2004 

Analyzing the Results and Writing the 
Final Report 

Synthesis of Seven Key Principles. 
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